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Abstract 

In the past few years, there has been an increasing interest in Massive Open Online Courses 
(MOOCs) as a new form of Technology-Enhanced Learning (TEL), in higher education and 
beyond. Recognizing the limitations of standalone MOOCs, blended MOOCs (bMOOCs) that aim 
at bringing in-class (i.e. face-to-face) interactions and online learning components together have 
emerged as an alternative MOOC model of teaching and learning in a higher education context. In 
this paper, we present the design, implementation, and evaluation details of a bMOOC course on 
“Teaching Methodologies” at Fayoum University, Egypt in cooperation with RWTH Aachen 
University, Germany, provided using the bMOOC platform L 2 P-bM 00 C. In order to gauge the 
usability and effectiveness of the course, we employed an evaluation approach based on Conole’s 
12 dimensions rubrics, ISONORM 9241/110-S as a general usability evaluation, and a custom 
effectiveness questionnaire reflecting the different MOOC stakeholder perspectives. 
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Usability; Quality assurance; Effectiveness 


69 







A Usability Evaluation of a Blended MOOC Environment: An Experimental Case Study 
Yousef, Chatti, Schroeder, and Wosnitza 

Introduction 

The emergence of Massive Open Online Courses (MOOCs) as a new Technology Enhanced 
Learning (TEL) model has the potential to change the existing higher education landscape. 
MOOCs have the opportunities of opening up learning and offering a wide range of choice in 
different areas and disciplines for a massive number of participants from anywhere all over the 
world to attend free online courses without any admission requirements (Liyanagunawardena, 
Adams, & Williams, 2013). Furthermore, MOOCs support a movement toward a vision of lifelong 
and on-demand learning for those who are working full time or have taken a break from formal 
education (Kop, Fournier, & Mak, 2011). Nevertheless, MOOCs suffer from several limitations. 
Yousef, Chatti, Schroeder, Wosnitza, & Jakobs, (2014a), for instance, provided an extensive 
review of the MOOC literature and stressed that the initial vision of MOOCs that aims at breaking 
down obstacles to education for anyone, anywhere and at any time is far away from the reality. In 
fact, most MOOC implementations so far still follow a top-down, controlled, teacher-centered, 
and centralized learning model. Attempts to implement bottom-up, student-centered, really open, 
and distributed forms of MOOCs are the exception rather than the rule. Other limitations of 
MOOCs include pedagogical problems concerning assessment and feedback (Hill, 2013), the lack 
of interactivity between learners and the video content (Griinewald, Meinel, Totschnig, & 
Willems, 2013), as well as high drop-out rates in average of 95% of course participants (Yousef, 
Chatti, Schroeder, Wosnitza, 2014b). A possible reason for the latter problem is the complexity 
and diversity of MOOC participants. This diversity is not only related to the cultural and 
demographic attributes, but it also considers the diverse motives and perspectives when enrolled 
in MOOCs. Furthermore, a major problem with MOOCs is the ignorance of the importance and 
benefits of face-to-face communication (Hollands & Tirthali, 2014; Schulmeister, 2014). Bill 
Gates, for instance, supports the idea of applying MOOCs in a blended-learning approach. He 
emphasizes the important role of the face-to-face interaction in didactical meta-communication 
(Young, 2012). These limitations raise some serious concerns on what role MOOCs should play, 
or how they should fit into the higher education landscape as an alternative mode of teaching and 
learning and a substantial supplement. 

On the way to address these limitations, the new design paradigm of blended MOOCs (bMOOCs) 
that aim at bringing in-class (i.e. face-to-face) interactions and online learning components 
together as a blended environment can resolve some of the hurdles facing standalone MOOCs 
(Bruff, Fisher, McEwen, & Smith, 2013). In fact, the bMOOCs model has the potential to foster 
student-centered learning, provide effective assessment and feedback, support the interactive 
design of the video lectures, consider the different patterns of participants in the MOOC, as well 
as bring the benefits of face-to-face interactions into the MOOC environment. Driven by these 
opportunities, this paper presents the design, implementation, and evaluation details of a 
bMOOC course on “Teaching Methodologies” at Fayoum University, Egypt in cooperation with 
RWTH Aachen University, Germany. The main goals of this study are: 
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1. Develop design patterns for effective bMOOC environments. 

2. Evaluate the developed bMOOC environment in terms of usability and effectiveness. 

3. Propose recommendations for the enhancement of the bMOOC environment based on 
the participants’ feedback. 


Blended MOOCs 

MOOC providers have already piloted the bMOOC concept within a higher education context. In 
particular, San Jose State University (SJSU) partnered with the Harvard and MIT non-profit 
MOOCs platform edX in the fall of 2012 to provide a bMOOC pilot experiment based on the 
“Circuits and Electronics” edX course. 87 SJSU on-campus students watched the MOOC video 
lectures on their own. Then, they practiced problems as homework. Afterwards, they met the 
faculty professor during class time to discuss the concepts presented in the video lectures. 
Meanwhile, they worked on projects in small groups and answered quizzes to check their learning 
progress. This bMOOC achieved a high success rate with 90% of the students passing the final 
exam, as compared with 55% in the traditional class of the previous year (Ghadiri, Qayoumi, 
Junn, Hsu & Sujitparapitaya, 2013). Even though the overall feedback showed positive results, 
there were some open issues, such as the lack of interaction between students and the video 
content as well as the lack of integration between the MOOC platform and the campus Learning 
Management System (LMS). Furthermore, the course was scheduled and led by the faculty 
professor and the students didn’t get the opportunity to engage in a self-organized learning 
experience. Therefore, they were more involved in the class time activity than the online practice 
on the edX platform. 

bMOOCs are still in the experimentation stage. Different approaches to design and embed 
bMOOC environments in the higher education landscape have been proposed in the MOOC 
literature (Bruff et al., 2013; Ghadiri et ah, 2013; Ostashewski, & Reid, 2012). These approaches, 
however, still follow a teacher-centered model. Recognizing the potential of bMOOCs to support 
new pedagogies such as self-organized and network learning, we focus in this study on learner- 
centered bMOOCs by providing a bMOOC environment where learners can take an active role in 
the management of their learning activities. 
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Blended MOOC Design 


As outlined in the introduction section MOOCs suffer from several limitations, namely following a 
teacher-centered and centralized learning model, the lack of effective assessment and feedback, 
the lack of interactivity between learners and the video content, the diversity of MOOC 
participants, and the absence of face-to-face interaction. In order to address the diversity issue in 
MOOCs, we analyzed and clustered the interest patterns of MOOCs stakeholders to create a 
meaningful picture of the MOOC community. Our main finding was a set of eight clusters of 
MOOC stakeholder perspectives namely, blended learning, flexibility, high quality content, 
instructional design & learning methodologies, lifelong learning, network learning, openness, and 
student-centered learning (Yousef, Chatti, Wosnitza, Schroeder, 2015). Table 1 illustrates the 
degree of support of the eight MOOC stakeholder perspectives in cMOOC, xMOOC, and face-to- 
face learning environments. None of these environments provides a full support for all MOOC 
stakeholder perspectives. 

Table 1 

Stakeholder Perspectives in cMOOC, xMOOC, and Face-To-Face Learning Environments 


Clusters 

cMOOCs 

xMOOCs 

Face-to-Face 

Blended Learning 

(V) 

(V) 

(V) 

Flexibility 

V 

(V) 

- 

High Quality Content 

- 

V 

- 

Instructional Design and Learning 
Methodologies 

- 

V 

V 

Lifelong Learning 

V 

V 

(V) 

Network Learning 

V 

(V) 

- 

Openness 

V 

(V) 

- 

Student-Centered Learning 

V 

- 

- 


V Completely (V) Partly - Very limited support 


An effective bMOOC that has the potential to take into account the different MOOC stakeholder 
perspectives can be viewed as the convergence of cMOOC, xMOOC, and face-to-face learning 
models, as depicted in Figure 1. In fact, cMOOCs support flexibility and openness and provide 
space for self-organized and networked learning where learners can define their own objectives, 
present their own view, and collaboratively create and share knowledge. xMOOCs focus on high 
quality content and follow a clear instructional design approach, where learning objectives are 
well-defined by teachers through short video lectures, often followed by e-assessment tasks. Face- 
to-face learning provides a number of benefits including direct feedback, coaching, and 
scaffolding. 
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xMOOC 


High quality content 
Clear instructional design 
Short videos lectures 
Weekly activities 
E-Assessment 


Face-to-Face 


• Scaffolding 

• Human interaction 

• Direct feedback 

Figure l. bMOOC as the convergence of cMOOC, xMOOC, and face-to-face learning. 


Blended MOOC Design Criteria 


We conducted a thorough literature review to collect a set of design criteria related to each cluster 
of MOOC stakeholder perspectives (Yousef et ah, 2014a; Yousef, Chatti, Schroeder, 2014c). 
Furthermore, we conducted a study to collect feedback from different MOOC participants 
concerning the importance of the collected criteria for each cluster (Yousef et al., 2014b). The 
highly ranked criteria related to each cluster are summarized in Table 2. 

Table 2 

Design Criteria ofbMOOCs 


Cluster 

Design Criteria 

Mean 

SD 


MOOCs system should provide coaching and scaffolding at 

critical times. 

4-50 

0.68 

Blended Learning 

Using video-conference tools to allow learners from 

different locations to communicate with the teachers. 

4.28 

0.84 


On-line participants list should be available to help learners 

to do synchronous discussions. 

4-13 

0-93 
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Provide at least two different times for learners to 

participate in the video-conference discussion. 

4.06 

0-95 

Flexibility 

Control features for video clip where appropriate, for 

example, play, repeat, full screen, slowdown, stop and 

pause. 

4.70 

0-53 

Write down the video keywords to help learners search for 

related videos. 

4.20 

0.92 

Provide links to videos encoded for different connection 

speed as much as possible. 

4-30 

0.84 

Provide related videos. 

4.07 

0.85 

Give learners examples that can be understood by everyone 

regardless of the cultural background 

4.08 

0.87 

High Quality 
Content 

Sound should be clear (even experienced presenters are 
prone to gabble when being recorded). 

4.81 

0.44 

The level of detail provided about the subject should meet 

the level of audience for which the resource has been 

designed. 

4-52 

0.68 

Offer references for facts and information in the video¬ 
lecture. 

4-39 

0.81 

Provide a summary of the video lecture. 

4-31 

0.86 

Synchronization of video and lecture note 

4-15 

0.94 

Provide a transcript of the video lecture. 

4.24 

0.94 

A different color can highlight pieces of information that 

are considered important. 

4.01 

1.00 

Provide integrated assessment within each task. 

4.12 

1.05 

Use short video clips, No more than 20 minute clips. 

4.29 

0-95 

Instructional 
Design and 
Learning 
Methodologies 

Objectives should be clearly defined at the beginning of 

each lecture. 

4-63 

0.69 

Offer course outline that contains objective, subject list and 

time schedule. 

4-50 

0.79 

Offer the course progress time line in visualization graphs. 

3-91 

0.91 

Each short video lecture should cover at most three 

objectives. 

4-34 

1.18 

Each quiz should give feedback and/or show the correct 

answers. 

4-57 

0.90 

Providing quiz - test report for learners to know their 

performance. 

4-49 

0.91 

Using different types of questions (e.g. short answers, 

essay, matching, multiple choice question and true/false 

4-44 

0.79 
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question). 



Using of electronic assessment such as (E-test, short 

quizzes and surveys). 

4.28 

0.78 

Long sentences, which normally contain conditional 

clauses, are difficult to understand. So convert every long 

sentence into two or more short ones. 

3-94 

1.04 

Starting videos with surprise information to attract the 

learners. 

3-73 

1-03 

Lifelong Learning 

Using English language for MOOCs to meet the wide range 

of learners from different countries and cultures. 

3-89 

1.02 

Using the international time [UTC] for deadlines and 

calendar. 

3-8o 

1.11 

Offer a subscribe feature to get videos and discussions 
updates. 

4.14 

0.88 

Video lecture should be tagged / categorized to enable 

easier search. 

445 

0.72 

Network Learning 

Supporting the collaborative learning among learners. 

4-52 

0.78 

Provide collaborative discussion tools. 

4-50 

0.69 

Provide e-mail notification. 

443 

0.84 

Offer notification tool for the important news and 

deadlines. 

4.41 

0.79 

Provide collaborative video annotation tools. 

3-93 

0.92 

Video platform should provide ranking tools “like & 

dislike”. 

348 

1-25 

Openness 

Provide a search box function to help learners to find 
different learning materials. 

4-51 

0.76 

Student can download the video lecture in their own 

devices. 

443 

0.89 

Use social networking tools to share learning materials. 

3-72 

1.22 

Student-Centered 

Learning 

Providing opportunities for learners to become more self- 

organized. 

4-31 

0.81 

Let the learners responsible for obtaining the objectives, 
have a voice in setting them. 

3-13 

1.15 

Allow learners to suggest new questions. 

3-93 

1.06 
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L 2 P-bMOOC Implementation 


The design criteria collected in Table 2 have built the basis for the implementation of the L 2 P- 
bMOOC platform on top of the L 2 P learning management system of RWTH Aachen University, 
Germany. L 2 P-bM 00 C represents a shift away from traditional MOOC environments where 
learners are limited to viewing video content passively towards a more dynamic and collaborative 
one. Learners are no longer limited to watching videos passively and are encouraged to share and 
create knowledge collaboratively. In L 2 P-bM 00 C, video materials are represented, structured, 
and collaboratively annotated in a mind-map format. 


The workspace of L 2 P-bM 00 C consists of an unbound canvas representing the video map 
structure of the lecture, a course selection section, and a sidebar for new video node addition and 
editing of video properties as shown in Figure 2. Possible actions on a video node include video 
annotations, video clipping, social bookmarking, and discussion threads. 


Teaching Methodologies - bMOOCI 0 





m 








0 OtMXMKS 



Figure 2. L 2 P-bM 00 C Workspace. 


Video Annotations 

The annotation section of video nodes is displayed in a separate layer above the main page and 
can be opened by clicking the “Annotation icon @” attached to map nodes. It consists of three 
main blocks: Interactive timeline, list of existing annotations and creation form for new 
annotations (see Figure 3). 
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Figure 3. L 2 P-bMOOC Video Annotation Panel. 


The interactive timeline visualizing all annotations is located right under the video and is 
synchronized with the list of complete annotations. By selecting timeline items, users can watch 
the video directly starting from the part to which the annotation points to. The timeline range 
corresponds to video duration and can be freely moved and zoomed into. Timeline items also 
include small icons that help to distinguish three annotation types: Suggestion, Question and 
Marked Important. Moreover, learners can adjust their own learning processes according to their 
points of interest and discuss with text or attaching links of relevant materials and discussion 
threads. Learners can also, insert new annotations while the video is in play mode at the current 
playback position. Furthermore, if learners believe the annotation contains an interesting or 
important note they have the option to “Like” it and later filtering items based on the number of 
likes. The “Trash” icon situated on top right corner of annotations is used to remove it. However, 
each item can be deleted only by its author. 

Search and Sort Functionalities 

Due to the long list of existing annotations in MOOC context, learners can perform searching and 
sorting actions. By entering a specific keyword, user name or annotation type, users can search for 
items in the list and a set of matching items will be drawn along with an updated interactive 
timeline. Sorting can be done based on date, time on video, rating or number of replies each 
annotation received. 
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Video Clipping 

In order to respond to the learners’ interest in a specific section of the video lecture, L 2 P-bM 00 C 
provides a clipping option that creates a new node representing a specific segment of the video. 
Clipping videos is supported for both complete and already clipped videos. In addition, these 
clipping videos can be accessed and annotated by all course participants. 

Bookmarks and Discussion Threads 

The options of attaching links of relevant materials and discussion threads are applicable for the 
original video lecture as well as the video nodes. Bookmarks represent online resources that can 
be added by all course participants and ordered based on their rating. They can be displayed in a 
separate JQuery Light box appearing on top of the application page. In contrast to annotations, 
discussion threads do not refer to any specific time in the video and may be used by course 
participants to discuss questions or suggestions relating to the general concept that the video 
node represents. 


Evaluation and Discussion 

In this study, we used the L 2 P-bM 00 C platform to offer a bMOOC on “Teaching Methodologies” 
at Fayoum University, Egypt in co-operation with RWTH Aachen University, Germany. We 
conducted a thorough evaluation of this bMOOC to gauge its usability and effectiveness. To 
achieve this, a user study was performed with the aim to gather quantitative and qualitative data 
from participants’ experience in this course. 

Evaluation Methodology 

We employed an evaluation approach based on Conole’s 12 dimensions rubrics, ISONORM 
9241/110-S as a general usability evaluation, and a custom effectiveness questionnaire reflecting 
the different MOOC stakeholder perspectives. 

Conole’s 12 Dimensions Rubrics 

Grainne Conole developed a new classification for MOOCs as part of the EFQUEL MOOC Quality 
Project (Conole, 2013). Conole’s evaluation rubric consists of the 12 dimensions, namely, level of 
openness, degree of massiveness, the amount of use of multimedia, the use of communication 
tools, the degree of collaborative learning, the type of learner pathway (i.e. learner-centered 
learning against teacher-centered learning), quality assurance, amount of reflection, assessment 
strategies, learning model (i.e. formal and informal), autonomy, and diversity (Conole, 2013). We 
evaluated the bMOOC against these 12 dimensions by following a three levels scale (i.e. low, 
medium, high), as shown in Figure 4. 
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Openness 

Massiveness 

Use of multimedia 

Communication 

Collaboration 


Low 


Medium 


High 


Learning pathway 
Quality assurance 

Amount of reflection 

Assessment 


I 

I 


l 

T 

I 

4- 

I 

I 

I 

I 


Formal learning 


Autonomy 


Diversity 


Figure 4. Evaluation of the bMOOC based on Conole’s 12 Dimensions Rubrics. 


The evaluation above shows the main characteristics of the “Teaching Methodologies” bMOOC. 
The course was offered through the L 2 P-bM 00 C platform hosted at RWTH Aachen University. It 
took place during the summer semester 2014 with duration of eight weeks. It was offered both 
formally to students from Fayoum University and informally with open enrollment to anybody 
who is interested in teaching methodologies. The teaching staff is composed of one professor and 
one assistant researcher from Fayoum University as well as one assistant researcher from RWTH 
Aachen University. A total of 128 participants completed this course. 93 are formal participants 
who took the course to earn credits from Fayoum University. These participants were required to 
complete the course and obtain positive grading of assignments. The rest were informal 
participants who didn’t attend the face-to-face sessions. They have undertaken the learning 
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activities at their own pace without receiving any credits. The teaching staff provided 6 video 
lectures and the course participants have added 27 related videos. This course was taught in 
English and participants were encouraged to self-organize their learning environments, present 
their own ideas, collaboratively create video maps of the lectures, and share knowledge through 
social bookmarking, annotations, forums, and discussion threads. 


General Usability Evaluation (ISONORM 9241/110-S) 

The ISONORM 9241/110-S questionnaire was designed based upon the International Standard 
ISO 9241, Part 110 (Priimper, 1997). We used this questionnaire as a general usability evaluation 
for the L 2 P-bM 00 C platform. It consists of 21 questions classified into seven main categories. 
Participants were asked to respond to each question scaling from (7) a positive exclamation and 
its mirroring negative counterpart (1). The questionnaire comes with an evaluation framework 
that computes several aspects of usability to a single score between 21 and 147. A total of 50 
questionnaires were completed. The table below illustrates the summary of the ISONORM 
9241/110-S usability evaluation. 

Table 3 

ISONORM 9241/110-S Evaluation Matrix (N= 50) 


Factor 

Aspect 

Mean 

Sum 

Suitability for tasks 

Integrity 

4.8 

14.4 

Streamlining 

5-1 

Fitting 

4-5 

Self- descriptiveness 

Information content 

4-9 

14-5 

Potential support 

4-8 

Automatic support 

4-8 

Conformity with user expectations 

Layout conformity 

5 

14-5 

Transparency 

4-8 

Operation conformity 

4-7 

Suitability for learning 

Learnability 

5-2 

13-9 

Visibility 

4-4 

Deducibility 

4-3 

Controllability 

Flexibility 

4-9 

13-9 

Changeability 

4-5 

Continuity 

4-5 

Error tolerance 

Comprehensibility 

2.4 

74 

Correct ability 

2-5 

Correction support 

2-5 

Suitability for individualization 

Extensibility 

4-8 

14.7 

Personalization 

5 

Flexibility 

4-9 

ISONORM score 

93-3 
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The majority of respondents were in the 18-24 age range. Female respondents formed the 
majority (90%). Participants have a high level of educational attainment: 70% of participants are 
Bachelor students at Fayoum University and 30% have a Bachelor’s degree or higher. They also 
have an experience with TEL courses. Nearly 75% reported that they attended more than two TEL 
courses. 

The overall ISONORM 9241/110-S score from the questionnaires was 93.3, which translates to 
“Everything is all right! Currently there is no reason to make changes to the software in regards of 
usability” (Priimper, 1997). In particular, suitability for individualization category was rated the 
best. This indicates that the participants had no issues with the adaptation of the bMOOC 
environment to fit their needs and preferences. One unanticipated finding was that the error 
tolerance category was rated the worst with a sum of 7.4, which indicates that participants had 
some issues in handling the system errors. 

In general, the ISONORM 9241/110-S evaluation results reflect a user satisfaction with the 
usability of the L 2 P-bMOOC platform. There is, however, still room for further improvement, 
especially in the error tolerance category. A possible enhancement of L 2 P-bMOOC would be to 
add a help guide (e.g. FAQs and system entry errors) as well as a video tutorial explaining the 
different features of the platform to ensure a better learning experience. 

Effectiveness Evaluation 

As stated above, learners have different goals when participating in MOOCs. The result of our 
study on diversity in MOOCs was a set of eight clusters of MOOC stakeholder perspectives. These 
include blended learning, flexibility, high quality content, instructional design & learning 
methodology, lifelong learning, network learning, openness, and student-centered learning 
(Yousef et al., 2015). The effectiveness evaluation in this paper aims at assessing whether these 
goals have been met in the offered bMOOC. 

There have been several attempts to evaluate the effectiveness of MOOCs. However, most of these 
studies only focus on a particular aspect of MOOCs. For instance, from a pedagogical perspective, 
Fini (2009) and Siemens (2013) focused on the effectiveness of cMOOCs for enhancing learning 
in the digital age. McAuley, Stewart, Siemens, & Cormier (2010) as well as Ostashewski and Reid 
(2012) focused on the effectiveness of the MOOC design, from a technical perspective. Our study 
aims at a comprehensive evaluation of MOOCs from different perspectives. We applied a multi¬ 
level effectiveness evaluation of the bMOOC that considers the different patterns of MOOC 
stakeholder perspectives. We designed a questionnaire to gauge whether the different goals of the 
bMOOC participants have been achieved, as shown from Table 3 to Table 10. The content of this 
questionnaire is based on relevant literature (Shee & Wang, 2008; Chang, 1999; Tobin, 1998). A 
5-point Likert scale was used from (1) strongly disagree to (5) strongly agree. 
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We defined a set of questions for each cluster. In order to ensure the relevance of these questions, 
we sent this questionnaire to a small panel of 5 learners as well as 5 learning technologies experts. 
They were asked for their opinions and suggestions for revising the questionnaire. Their feedback 
included a refinement of some questions and shifting questions to other clusters. The revised 
questionnaire was then given to the bMOOC participants. The following sections present the 
results of the effectiveness evaluation of the bMOOC. 

Internal Course Diversity 

We started our questionnaire by asking participants about the purpose of their participation in 
the Teaching Methodologies bMOOC, based on the eight clusters of MOOC stakeholder 
perspectives outlined above. The participants had the possibility to select more than one answer. 
Figure 5 shows the summary of their responses. The results reflect diversity in the participants’ 
perspectives. 


Blended Learning 
Flexibility 
High Quality Content 
Instructional Design and Learning Methodologies 

Lifelong Learning 
Network Learning 
Openness 
Self-organized Learning 



Figure 5. What is the purpose of your participation in this course? (N= 50) 


Blended Learning 

The design of blended learning environments bringing together face-to-face and online learning 
can be a flexible and effective model to enhance classroom learning and to improve relationships 
with teachers and peers (Bruff et al., 2013). The course participants were asked to watch the 
lecture videos online and use the L 2 P-bM 00 C platform to collaboratively annotate and discuss 
the lecture content. The face-to-face sessions are then used to elaborate more on the concepts 
presented in the video lecture, discuss practical aspects of the course, and provide direct feedback 
to the group projects. 

Table 4 lists the five evaluation items of the blended learning category. The agreeability mean of 
the respondents is quite high at 4.4. Item 2 “Bringing together face-to-face and online learning 
increases my motivation to share and discover new ideas” obtained the highest mean score of 4.5, 
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which indicates that the bMOOC increased the course participants’ motivation. The participants 
reported that the permanent coaching and scaffolding provided by the teachers, as well as the 
continuous direct feedback from other course participants had positive impact on their 
motivation in the course. Moreover, the face-to-face interactions with participants with diverse 
backgrounds and interests increased their engagement and trust. This reveals the importance of 
the human factor in bMOOCs. This is consistent with the findings of Bruff et al. (2013) who 
pointed out that bMOOCs can improve the learning outcome, because participants in bMOOCs 
can benefit from the opportunities for independent learning, increased engagement and 
motivation, and flexibility of bMOOCs. 

Table 4 

Descriptive Results of Blended Learning (N=so). 


No 

Blended Learning 

Evaluation Item 

M 

SD 

1 

Bringing together face-to-face and online learning helps me to 
improve my academic achievements outcome. 

4-3 

0.74 

2 

Bringing together face-to-face and online learning increases my 
motivation to share and discover new ideas. 

4-5 

0.76 

3 

Bringing together face-to-face and online learning enables me to 
accomplish tasks more quickly. 

44 

0-73 

4 

Blended learning approach can be used to supplement traditional 
classroom approach. 

44 

0.70 

5 

I felt a sense of satisfaction about this blended learning 
environment. 

4-3 

0.58 

Blended Learning Average 

44 

0.70 


Flexibility 

One of the successful factors in MOOCs is flexibility (Mackness, Mak, & Williams, 2010). The six 
evaluation items in Table 5 aim at assessing the flexibility level of the bMOOC. Most participants 
reported a high satisfaction with the diversity of the provided learning materials as well as the 
ability to access the learning resources at any time and from anywhere. 
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Table 5 

Descriptive Results of Flexibility Level (N=so) 


No 

Flexibility 

Evaluation Item 

M 

SD 

1 

I can access the learning activities at any time convenient to me. 

4-4 

0.63 

2 

I can access to lectures and learning activities from anywhere. 

44 

0.67 

3 

The learning environment provides me a wide range of materials 
that I can choose from. 

4-3 

0.85 

4 

I was able to access the learning materials without much difficulty. 

4.6 

0.70 

5 

The video mind-map content makes me want to explore the course 
further. 

4.2 

0.62 

6 

The learning environment allows me to focus on the learning 
activities suitable to me. 

44 

0.79 

Flexibility Average 

44 

0.71 


High Quality Content 

One of the most important factors to empower and engage learners around the world to 
participate in MOOCs is the quality of course content (Yousef et al., 2015). Shee and Wang (2008) 
pointed out that learners place great value on online courses where the content is well-organized, 
interactive, the presentation of the subject is clear, and in the right length. The six evaluation 
items in Table 6 aim at measuring the quality of the content in the provided bMOOC. The mean 
score in this category was 4.4. 
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Table 6 

Descriptive Results of High Quality Content (N=so). 


No 

High Quality Content 

Evaluation Item 

M 

SD 

1 

The presentation of the subject content is clear. 

44 

0.74 

2 

The video-map helps to structure the learning content. 

4-5 

0.71 

3 

The interactive video annotations help to improve the quality of 
the learning content. 

44 

0.86 

4 

The information that has been presented in the discussions helps 
me to better understand this course. 

44 

0.86 

5 

The feedback on my annotations helps me to reflect on the course 
content. 

4-3 

0-93 

6 

Browsing the bookmarked articles on each video-node helps me to 
better understand the learning content. 

4-5 

0-73 

High Quality Content Average 

44 

0.81 


Most respondents agreed that the course materials and the user-generated content (e.g. mind 
maps, discussions, annotations, bookmarks) were very helpful to better understand the course 
concepts. In particular, browsing highly rated bookmarked articles on each video node and 
receiving comments and suggestions on the annotations helped to improve the quality of the 
course content. 

Instructional Design and Learning Methodology 

Effective instructional design and learning methodology can make bMOOCs more attractive and 
motivating (Yousef et al., 2015). Table 7 illustrates the evaluation of the effectiveness of the 
instructional design and learning methodology used in this bMOOC. Respondents were generally 
positive regarding the well-defined objectives, the clear structure, the effective tools, and the 
teaching assistance offered to support the learning activities in this course. One unanticipated 
finding was that the tutor feedback on the assignments obtained a relatively low mean score of 4. 
Possible reasons for this might be the limited time of the teaching team and using only one type of 
assessment, namely teacher assessment. Indeed, the ability to evaluate a large number of learners 
in MOOCs is a highly challenging task. It is necessary to go beyond traditional teacher assessment 
methods and apply open assessment methods that fit better to the bMOOC environments 
characterized by openness, networking, and self-organization. These include peer-assessment, 
self-assessment, and e-assessment methods (Yousef et al., 2014b). 


This work is licensed under a Creative Commons Attribution 4.0 International License. 


85 



A Usability Evaluation of a Blended MOOC Environment: An Experimental Case Study 
Yousef, Chatti, Schroeder, and Wosnitza 


Table 7 

Descriptive Results of Instructional Design and Learning Methodology (N=so) 


No 

Instructional Design & Learning Methodology 

Evaluation Item 

M 

SD 

1 

The learning objectives are clearly stated in each lecture. 

4.2 

0.89 

2 

The scope of the lecture is clearly stated. 

4-4 

0.84 

3 

The structure of this course keeps me focused on what is to be 
learned. 

4-5 

0-73 

4 

I always know where I am in the course. 

4-4 

0.83 

5 

The various tools in this learning environment are effective. 

4-4 

0.64 

6 

I have the possibility to ask my tutor what I do not understand. 

44 

0.83 

7 

The tutor responds promptly to my queries. 

4.2 

0-57 

8 

I can approach the teaching team in this course when needed. 

4.6 

0-54 

9 

The assessment in this course enhances my learning process. 

4-3 

0-53 

10 

The tutor sends me comprehensive feedback on my assignment. 

4 

0-95 

11 

The grading criteria were clearly communicated at the beginning 
of the course. 

4-3 

0-73 

Instructional Design & Learning Methodology Average 

4-3 

0-73 


Lifelong Learning 

Learning is no longer restricted to the formal higher education context. MOOCs are providing a 
disorganized and unstructured learning model for informal participants. This kind of learning 
tends to be experimental, spontaneous and free from rigid curricula. There is a wide agreement 
among MOOC providers and researchers that MOOCs open doors for new opportunities for 
lifelong learning outside the boundaries of formal educational institutions (Milligan & Littlejohn, 
2014; Kizilcec, Piech, & Schneider, 2013; Kop et al., 2011). Several studies on the profile of MOOC 
participants found that the majority has a Bachelor or a Master degree and in most of the cases 
the MOOC is used for job (re)training and lifelong learning purposes (Christensen, Steinmetz, 
Alcorn, Bennett, Woods, & Emanuel, 2013; Kizilcec et al., 2013; Kop et al., 2011). This is quite 
different in bMOOCs, as the majority of participants take the MOOCs as part of a university 
credit-bearing course. In our study, only 30% of the course participants are lifelong learners 
tending to learn through this bMOOC for their personal or professional interest rather than 
obtaining an official academic degree. As shown in Table 8, most of the respondents agreed that 
the course helps them improve skills required for their future job as school teachers and opens 
new opportunities to advance their knowledge and expertise. This confirms the potential of the 
bMOOC to support lifelong learning activities. The findings of the current study are consistent 
with those of Milligan and Littlejohn (2014) who emphasize the important role of MOOCs for 
opening up, supporting and enabling professional learning, allowing opportunities to link formal 
and informal learning. 
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Table 8 

Descriptive Results of Lifelong Learning (N=so) 


No 

Lifelong Learning 

Evaluation Item 

M 

SD 

1 

The course helps me to improve skills required for my (Future) 
job. 

4.6 

0.69 

2 

The learning environment encourages me to invite participants 
from outside the university. 

4-3 

0.71 

3 

I will use this learning environment frequently for my 
continuous learning in the future. 

4.6 

0.70 

4 

Courses are delivered at suitable time for professional 
participants (workers). 

4-3 

0.80 

5 

The course content is also suitable for professional participants 
(workers). 

4-5 

0.50 

6 

This learning environment opens new opportunities to advance 
my knowledge and expertise. 

44 

0-73 

Lifelong Learning Average 

4-5 

0.69 


Network Learning 

Network learning is important in open and distributed learning environments like bMOOCs 
(Chatti, Schroeder, Jarke, 2012). A set of seven items for the evaluation of the offered bMOOC in 
terms of collaborative and network learning are shown in Table 9. 

Table 9 

Descriptive Results of Network Learning (N=so) 


No 

Network Learning 

Evaluation Item 

M 

SD 

1 

I can interact with other students and the tutor synchronously 
and asynchronously. 

44 

0-54 

2 

I am allowed to create and manage my own group. 

4-5 

0.82 

3 

It is easy to work collaboratively with other students involved in 
a group project. 

44 

0.74 

4 

The communication tools enhance my interaction and 
collaboration with my course mates. 

4.6 

0-54 

5 

I was supported by positive attitude from my course mates. 

44 

0.86 

6 

I share what I have learned in this course with others outside of 
the learning environment. 

44 

0-73 

7 

The learning environment helps me receive support and 
feedback from other participants. 

44 

0.88 

Network Learning Average 

44 

0-73 
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In this category, the high mean average of 4.4 indicates the effectiveness of the bMOOC in 
supporting network learning. In fact, the participants agreed that the collaboration and 
communication possibilities offered in L 2 P-bM 00 C (i.e. group workspaces, discussion forums, 
live chat, social bookmarking, and collaborative annotations) allowed them to share, discuss, 
exchange, and collaboratively construct knowledge as well as receiving feedback and support from 
peers. 

Openness 

Openness is one of the characteristics in MOOCs. It refers to providing a learning experience to a 
vast number of participants around the globe regardless of their location, age, income, ideology, 
and level of education, without any entry requirements, or course fees to access high quality 
education. Openness also refers to providing open educational resources (OER) following the 
4 Rs, namely Reuse, Revise, Remix, and Redistribute (Peter & Deimann, 2013). Most of the 
MOOCs on the market are open for participants without any admission requirements and for free. 
They are, however, not open from a copyright perspective. For instance, Coursera does not permit 
users to reproduce, retransmit, distribute, or publish any material from its platform. 

The offered bMOOC does not only enable participants to register for the course for free and 
without any academic requirements, but also enable them to reuse, revise, remix, and redistribute 
all course materials as seen fit. Table 10 shows the high satisfaction of the respondents with the 
level of openness in the bMOOC. 

Table 10 

Descriptive Results of Openness (N=5o) 


No 

Openness 

Evaluation Item 

M 

SD 

1 

I register for this course free of charge 

4-7 

0.47 

2 

There were no academic requirements for registration. 

4-5 

0.99 

3 

The learning material is available for free download. 

4.6 

0.48 

4 

This learning environment enables me to adapt the learning 
material to better meet my needs. 

4.6 

0.72 

5 

I can reuse the learning materials from this course to produce 
my final report assignment. 

4-5 

0.81 

Openness Average 

4.6 

0.69 


Self-Organized Learning 

One important goal of participation in MOOCs is self-organized learning. bMOOCs can provide a 
space for learners to be active participants in the learning process and to get mutual support 
(Chatti, 2010). Table 11 shows the results of ten evaluation items to examine how much the 
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bMOOC supports self-organized learning. The mean average was 4 which indicate that a majority 
agreed that the learning environment allowed them to be self-organized in their learning process. 
In particular, the participants reported that the representation of the lecture in a mind map view 
and the video clipping feature helped them to learn independently from teachers. The results 
further confirm that the learning environment encourages participants to work at their own pace 
to achieve their learning goals and keep them in control of their learning progress. Items 5 and 10 
obtained the lowest mean score of 2.8 and 2.7, respectively. This shows that the participants had 
some difficulties in tracking and monitoring their learning activities and those of their peers. 
Further improvement should be done to address this important issue. This can be in the form of a 
learning analytics tool that enables to collect, visualize, and analyze the data from learning 
activities (e.g. comments, likes, newly added nodes) to support monitoring, awareness, self¬ 
reflection, and feedback (Chatti, Lukarov, Thus, Muslim, Yousef, Wahid, Greven, Chakrabarti, & 
Schroeder, 2014). 

Table 11 

Descriptive Results of Self-Organized Learning (N=5o) 


No 

Self-Organized Learning 

Evaluation Item 

M 

SD 

1 

I am allowed to create my own video mind-map. 

4-3 

0.81 

2 

I am allowed to work at my own pace to achieve my learning 
objectives. 

4-4 

0.60 

3 

I decide how much I want to learn in a given time period. 

4-5 

0.68 

4 

I decide when I want to learn. 

4.2 

0.78 

5 

I am aware of the activities of my peers in the course. 

2.8 

1.11 

6 

I have the possibility to ask other students what I do not 
understand. 

4-t 

0-73 

7 

I can organize my own learning activities. 

4-4 

0.64 

8 

I can learn independently from teachers. 

4-3 

0.69 

9 

I was in control of my progress as I moved through the 
material. 

4-4 

0-73 

10 

I can easily keep tracking of all activities (i.e. comments, likes, 
newly added nodes, etc.) in this course. 

2.7 

1-33 

Self-Organized Learning Average 

4 

0.81 


Conclusion 


Massive Open Online Courses (MOOCs) present an emerging branch of online learning that is 
gaining interest in the Technology-Enhanced Learning (TEL) community. Despite their 
popularity, current MOOCs suffer from several limitations. These include following a teacher- 
centered and centralized learning model, the lack of effective assessment and feedback, the lack of 
interactivity between learners and the video content, the diversity of MOOC participants, and the 
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absence of face-to-face interaction. In this paper, we argued that the blended MOOC (bMOOC) 
model has the potential to address these issues. The purpose of the current study was to design, 
implement, and evaluate a bMOOC course on “Teaching Methodologies” at Fayoum University, 
Egypt in cooperation with RWTH Aachen University, Germany, provided using the bMOOC 
platform L 2 P-bMOOC. In order to gauge the usability and effectiveness of the course, we 
employed an evaluation approach based on Conole’s 12 dimensions rubrics, ISONORM 9241/110- 
S as a general usability evaluation, and a custom effectiveness questionnaire reflecting the 
different MOOC stakeholder perspectives. The results of the study revealed a general satisfaction 
with the bMOOC in terms of usability and effectiveness. There was a wide agreement among the 
participants that offered bMOOC can address the limitations of MOOCs outlined above. The 
study, however, shows that it is crucial to investigate learning analytics techniques to foster 
monitoring, awareness, self-reflection, and feedback in bMOOC environments as well as to 
develop new assessment methods, such as peer-assessment, self-assessment, and e-assessment 
that reflect the open and massive nature of MOOCs. 
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